I actually tried using the categories from the AdvBench paper (profanity, graphic depictions, threatening behavior, misinformation, discrimination, cybercrime, and dangerous or illegal suggestions) but, honestly, the results aren't great: pretty much everything ends up under dangerous or illegal suggestions, which isn't super useful IMO 😅

{
    "0": ["cybercrime", "dangerous or illegal suggestions"],
    "1": ["cybercrime", "dangerous or illegal suggestions"],
    "2": ["dangerous or illegal suggestions"],
    "3": ["dangerous or illegal suggestions"],
    "4": ["cybercrime", "dangerous or illegal suggestions"],
    "5": ["dangerous or illegal suggestions"],
    "6": ["discrimination", "dangerous or illegal suggestions"],
    "7": ["dangerous or illegal suggestions"],
	...
}

So then I tested out that Collaborative, Human-Centered Taxonomy of AI, Algorithmic, and Automation Harms instead. It breaks things down into main categories (e.g.: Physical, Financial, Psychological) and subcategories (e.g.: Privacy Loss, Economic Instability, Coercion/manipulation), and honestly, the output looks a lot more meaningful. I think it captures nuance better instead of labelling everything into one or two types.
For now, I only categorized the first 50 prompts with the help of the Claude 3.7 model and I put the results in this file pyrit/datasets/harm_categories/adv_bench_dataset.json.

Well... I really hope I’m not making things more chaotic 😅 I just want to make sure we’re getting the best possible results!
Let me know what you think :)